In computer programming, by “unit testing” is understood a software verification and validation method for checking whether individual units of source code are fit for use. A “unit” is the smallest testable part of a software application. For example, in procedural programming a unit is an individual function or procedure. A unit test provides a strict, written contract that the unit under test must satisfy. The goal of unit testing is to show that the individual parts (units) of the application software are correct when ran in isolation. Usually, a unit test must satisfy code coverage criteria (often Statement or Branch).
Unit testing is considered a fundamental step for developing modern software of high quality.
A unit test suite is organized by means of attributes, assertions and constraints. The execution of a set of unit tests can be controlled by means of attributes. The assertions and constraints affect the calculation of individual test case verdicts (test case passes or test fails), attributes support, such as the organization of sets of test cases (called test suites) and the automatic provision of test data to test cases.
Currently, unit tests are always executed to the last test case even if errors or fail events occur or have occurred during the execution, where the errors or fail events suggest stopping the test suite immediately. Moreover, tests are executed in an undefined (implementation-specific) order. However, often after errors or fails, the remaining test cases may be invalid because they depend logically on test cases running before and obviously affect the final test outcome. Performing the complete test run translates into wasted time and blocks test machines, requires undue use of memory (for storing test results), or even leads to damage of test machines. Therefore, it is important to control the execution of sets of unit tests.
Moreover, it is important to control the execution order of test cases within a test suite and to solve the issue of test category combination. Test cases are executed sequentially, in parallel or randomly. Furthermore, the set of tests to be executed are restricted or extended in order to meet situation-dependent test goals and project constraints and hence also obey economic constraints goals.
While existing test automation frameworks, languages and associated tools support the development and automatic execution of single test cases, the support for organizing and executing test suites (each combining several test cases or complete test suites) is rather rudimentary. Up to now the unit test control issues listed above are usually approached by verbose and error-prone imperative programming, often in an ad-hoc manner.
Test developers skilled in the art of testing are used to program test cases but have problems programming test suites, i.e. selecting, sequencing and controlling test suites and performing overall test control. Most times, test developers mix statements controlling test cases with statements controlling test suites, leading therefore to test specifications that are hard to understand, verify, maintain and apply.
Because of the difficulties in selecting the proper set of test cases and test suites, almost all the available tests are selected for a test run. Each test case checks, usually at several places in the test code, whether the test case is applicable or not. These checks are test control code which are not mixed with other code. If mixed, a test specification that is hard to understand results.
It is of special importance how the test case verdicts (pass, fail) affect the test execution sequence. Currently, the only solution to this constraint is to have only one test case per test suite (TextFixture) and to sequence the test suites outside the unit test framework. This can be achieved manually or by a script (batch file) controlling the execution of arbitrary programs on the command line.
Manual execution of test suites is not a solution because, due to the huge amount of unit tests, the unit tests are executed automatically on a daily, weekly or other periodic basis, and approaching the test control issue with a script in one of the diverse scripting languages and language variants is a paradigm discontinuity, depends on the properties the unit test tool allows to control and observe at the proprietary tool's interface, and, perhaps most important of all, is to be done in an imperative manner instructing the test tool, i.e. test execution engine, step-by-step what to do.
It is also important to execute the test cases and test sets in logical sequence. Currently this constraint is often solved by naming conventions with the intention to execute the test cases in alphanumeric order. However, there are no guarantees that different test execution engines or even two variants of the same execution engine consider the test names for scheduling (sequencing) the individual test cases.
The selection of test cases to be run by combining test categories is as well of importance. Currently it is not possible to combine test categories with unit test tools in a meaningful way. Only tests of one category can be executed at a time. Two or more test categories can be selected only in the sense of list concatenation with implementation-dependent execution sequence wherein the list order is undefined.